Repository navigation
feat(dynamo)!: serve NeMo RL's own vLLM rollouts through Dynamo's backend SDK - #4905
Conversation
…end SDK `policy.generation.dynamo: true` on a vLLM generation config is now the whole Dynamo switch. NeMo RL's own vLLM workers register their AsyncLLM with a managed dynamo.frontend through Dynamo's engine-agnostic backend SDK (dynamo.common.backend), from inside each worker's actor loop. No Dynamo-owned vLLM code runs, so refit, pause, quantization, speculative decoding and parallelism are the native backend's own, and a vLLM bump never waits on Dynamo. - One request pipeline (vllm/request_pipeline.py) serves native generate_async, the native OpenAI server and the Dynamo adapter: the sampling-profile check, clamps, R3, token fields, capture, prefix splice and media. - Rollouts go through the frontend's KV-aware router. The engine publishes KV events, and functional overlays fail if the router applies none. - NeMo-Gym calls go through the frontend in Gym's Dynamo dialect (NVIDIA-NeMo/Gym#4817), with engine-side token and media capture. - Startup rejects what Dynamo cannot serve: colocated generation, fleet health, the generation router, a bearer token, R3 with Gym but no capture, parser plugins and chat settings with no frontend counterpart. - Removed: backend: dynamo, DynamoGeneration, the dynamo.vllm launcher, its refit routes, token wrapper, metrics scraper and isolated venv. Old keys are rejected with the migration. - ai-dynamo and ai-dynamo-runtime 1.5.1 join the vllm extra, without the planner deps. BUILD_DYNAMO installs only etcd and nats-server. - CI: the Dynamo lane runs native SingleController tests as overlays: math, Gym token capture, reload refit, and R3 with deferred-route capture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Run and CI breaks: - Each engine publishes KV events on its own abstract ipc:// socket, so engines on one node can no longer draw the same port (C1). - Engines get an OS-assigned DYN_SYSTEM_PORT, so a frontend on another node can read a custom chat template (C2). - The Gym capture test skips without nemo_gym (C3). - Gym calls apply the generation config's max_tokens only when the request leaves it unset, as vLLM's server does (C4). - Lint: minimized swe1 recipe (C5) and import order (C6). Behavior: - vLLM client errors reach the client as InvalidArgument with their message, not as a bare 500 (C7). - Non-capture Gym calls return routed experts with R3 on, and Gym#4817 (now a369e28) lifts them, so R3 + Gym no longer needs token capture (C8). - A disconnected capture call fails instead of being staged (C9). - EP > TP is rejected at startup with Dynamo (C10). - The KV-event block size matches vLLM's main-attention kinds (C17). Messages and docs: timeout wording and error (C11, C12), missing etcd/NATS advice (C13), dependency bounds (C19) and ray.sub's port layout (C20). Tests: a Gym-without-capture Dynamo overlay in L1 (C14), the rollout retry loop (C15), media dispatch and failure mapping (C16), hybrid block size (C17) and the Gym engine_data key contract (C18). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
left a comment
There was a problem hiding this comment.
Summary
The design holds up, and most of the 20 inline comments are small. Four should be fixed before this leaves draft, because each breaks a run or CI:
- (C1) Engines on one node pick their KV-event port at random and can collide; the second engine then fails to start (24% of starts at 8 engines per node). An abstract
ipc://socket per engine removes the port. - (C2) A custom chat template stops a multi-node run from ever serving: the frontend cannot read a template file that exists only on the engine nodes.
DYN_SYSTEM_PORT=0lets each engine serve it. - (C3) The capture test imports
nemo_gymwithout a guard, so all threeL0_Unit_Tests_Vllm_*shards now fail at collection. - (C4) Gym calls are capped by the model's generation-config
max_new_tokenseven when they setmax_tokens, which vLLM's own server does not do.
Two more fail CI's lint as soon as it runs: C5 (configs-minimize-check) and C6 (import order). The rest are lower-severity bugs (C7, C9), usability asks (C8, C10, C12, C13), docs (C11, C19, C20) and test gaps (C14–C18).
What works well
dynamo: trueas a flag on the vLLM config, not a backend value, is the right shape. It cuts the"dynamo"literals outside the Dynamo package from 21 in 9 files to 5, and the conflicting settings are rejected at the one place every entrypoint calls (generation/__init__.py#L158-L169).- The request path shares code with native instead of copying it:
off_policy_sampling_error,chat_generation_tokensandchat_routed_experts,splice_prefix_tokenswith an EOS-count input, and_rollout_max_new_tokens/_rollout_sampling_params. Nativegenerate_asyncis about 230 lines shorter for it. - The Dynamo L1 tests are the native SingleController tests plus
++policy.generation.dynamo=trueand one shared check script, so Dynamo coverage moves with native coverage. - The decoupling guard: an import allow-list that runs in every lane (test_dynamo_vllm_decoupling.py#L63) and a fresh-interpreter "the SDK imports no vLLM" check in the vLLM lane (#L78). Both fail rather than skip.
- Narrow seams:
ManagedDynamoRuntimenow owns only the services and takes acheck_enginescallback, so a failed registration fails with its cause;ROLLOUT_ANNOTATIONsits driver-side, so the driver never imports ai-dynamo. - The engine tests drive a real vLLM worker over a fake
AsyncLLM, through the native helpers it shares, with no GPU. - All three critical fixes from the #4822 review are here and covered: the
top_kpin (reverting it fails a test), the f32 comparison of temperature and top_p, and the echoed-prefix splice.
PR description
- The "One request pipeline" bullet overstates
request_pipeline.py: it holds only the rollout result path (RolloutResult,selected_logprobs,rollout_result,rollout_outputand the wire codecs), and the native OpenAI server does not import it. Suggested text: "Shared code, not copies. Rollouts sharevllm/request_pipeline.py(result, R3 routes) and the worker's budget and sampling with nativegenerate_async. Gym calls reuse the native endpoint's sampling check, prefix splice, token-field and routed-expert helpers, and the worker's capture lifecycle." - (C19) "vLLM bumps are not blocked by Dynamo … unit guards enforce both" holds for code only. Dynamo's dependency bounds (
ai-dynamo-runtimecaps pydantic at 2.13) can still hold a vLLM bump until Dynamo releases or an override is added; say so here too.
Performance (no action)
At the scale of the validation runs the step times are mostly noise (one engine, 1 to 10 steps, and native's own repeats vary about 6x), so they are not worth a table. One fact is: the R3 routed-experts payload adds no visible step time (median exposed generation / step time over 10 steps: 1.08 s / 4.29 s native, 1.09 s / 4.28 s Dynamo). Throughput at scale stays the follow-up the PR already lists.
Legacy entrypoints (FYI, no action)
- (C8) R3 + Gym on
run_grpo_nemo_gym.pypasses startup and fails at the first prev-logprob step; C8's fix closes it. - An FP8 KV cache with
dynamo: trueonrun_grpo.pyorrun_ppo.pyfails with "Async rollouts is not supported with kv cache fp8 enabled" (grpo.py#L1719-L1725, ppo.py#L945-L947), because Dynamo turns the async engine on. - The swe1 Dynamo nightly runs the legacy
run_grpo_nemo_gym.pyon 3 nodes (grpo-nanov3-30ba3b-3n8g-megatron-dynamo-swe1.sh#L32), which no validation row covers at this head. Its reasoning parser now resolves tonemotron_v3instead of the oldnemotron_nano; the PR documents them as the same parser family (config.py#L92-L96).
Follow-ups (tracking, not for this PR)
- Move the vLLM worker's token-capture lifecycle into its own object (a
VllmRequestCapturein a newvllm/token_capture.py), as Megatron already has (megatron/token_capture.py#L14-L28). About 280 lines would leave vllm_worker_async.py, and the Dynamo engine's calls into private worker methods would drop from 10 to 3. - Session affinity covers only capture calls: Gym sends
x-dynamo-session-idonly from its capture handler (external_capture.py#L450-L455), although theng_prefixsplice on non-capture calls also happens after routing. Unmeasured; the fix is on the Gym side. - NeMo RL does not notice an engine that stops serving after registration:
dynamo_endpoint_erroris polled only during the registration wait. Dynamo logs the cause (etcd lease loss at ERROR, lease.rs#L45-L57), and the fleet just shrinks. A done-callback on the endpoint task that logs at ERROR would be cheap. - Upstream Dynamo issues to file:
Worker.run()callsconfigure_dynamo_logging()unconditionally (worker.py#L230), which removes the host process's root log handlers and sets the root level fromDYN_LOG, and its vLLM log formatter ignoresDYN_SDK_DISABLE_ANSI_LOGGING. Engines embedded in a host process need an opt-out.- A request with more than 4 stop strings gets a 500 instead of a 400, and the message says "less than 4" although 4 is accepted (openai.rs#L230-L233).
Generated by Claude Code
|
/ok to test 08c50d2 |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
/ok to test 2ba2cae |
Resolves the docker/Dockerfile conflict with NVIDIA-NeMo#4340 (uv cache mounts). The BUILD_DYNAMO RUN keeps this branch's form: docker/dynamo/install.sh now only downloads etcd and nats-server, so it is not a uv-writing RUN and takes no uv cache mount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
/ok to test b065eb6 |
Gym#4817 (jthomson04/dynamo-frontend-dialect) is rebased onto Gym main and now sits on top of it; its head moves from a369e28 to a5f936c. The only conflict was Gym main's new custom-tool-strategy validator, which sits next to the Dynamo frontend validator; both are kept. Gym's dependencies are unchanged, so uv.lock needs no change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
/ok to test 94c0e96 |
|
Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
Review
Request changes for C23: enabling Dynamo makes a valid multi-shard Gym configuration fail during shard planning. C24 fixes the local-install instructions, and C25 is a low-severity maintainability suggestion.
The shared native request pipeline is a useful seam to retain. The current source contains the earlier fixes for request-specific token limits, vLLM client-error mapping, disconnected captures, per-engine IPC endpoints, and engine system ports; those findings are not repeated.
This was a broad static review of the changed paths and their affected integration paths. Tests, linters, image builds, and GPU runs were not executed. The Gym gitlink was assessed, but its upstream contents were not populated in the mounted tree. The configuration/build topic was independently re-inspected because its delegate's completed report was unavailable.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
…osition From the NVIDIA-NeMo#4905 review: - C23: setup_nemo_gym_config wrote the dialect as a top-level policy_model overlay, which a sharded Gym plan rejects as unowned. Setup now stashes it under a NeMo RL key (dynamo_dialect, alongside tokenizer_config), and _build_gym_actor_config applies it to each actor's composed config. A shard's own template kwargs still win over the native defaults. - C24: the local install instructions export NEMO_RL_DYNAMO_BIN_DIR, so the run that follows finds etcd and nats-server. - C25: the two in-function rationale blocks in native_engine.py are one line each; the design doc carries the full reasons. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
/ok to test e182811 |
Resolve examples/configs/grpo_math_1B.yaml: keep main's new lora_refit_mode default and this branch's comment, which no longer lists the removed dynamo backend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
/ok to test b498510 |
C59. Dynamo's Worker.run() starts with configure_dynamo_logging(), which replaces the whole actor's Python logging: it removes NeMo RL's root handler, sets the root level from DYN_LOG (so NRL_LOG_LEVEL=DEBUG records are dropped), and swaps vLLM's handlers for an always-ANSI formatter. start_dynamo_endpoint now replaces that module global with a no-op before the Worker starts. A test checks the replacement keeps the root handlers, and another fails if Worker.run() stops calling the setup by that name, so a Dynamo bump cannot silently undo it. ai-dynamo/dynamo#18447 asks for a WorkerConfig opt-out that replaces this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
/ok to test c6a4c66 |
docker/Dockerfile conflicted with NVIDIA-NeMo#4863, which ran docker/dynamo/install.sh with a throwaway uv cache. On this branch install.sh only downloads etcd and nats-server, so the RUN keeps calling it directly. test_prune_uv_cache.py's Dockerfile test keeps its no-cache-mount check and drops the two lines that asserted the throwaway cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
/ok to test 4cbebb8 |
vllm_worker_async.py's import block and pyrefly.toml's project-includes conflicted with NVIDIA-NeMo#4506 (prefix recovery): each side added lines at the same spot. Both sides are kept. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
/ok to test 9919396 |
…-gym Brings in main's prefix recovery (NVIDIA-NeMo#4506) through NVIDIA-NeMo#4905. This branch moved the worker's inline chat server into vllm/chat_server.py, and NVIDIA-NeMo#4506 changed that inline server, so the worker's conflicts keep this branch's calls into chat_server and NVIDIA-NeMo#4506's server changes are applied there: - the request's restore fields, and on_sampling_params from the request class the routes bind to the worker; - set_max_tokens sets max_tokens when a request set neither field; the frontend's render lowers the budget only when the call set one; - preprocess_chat resolves a restored call's saved prefix. The block is three functions (resolve_restored_call, classify_restored_terminal, serve_restored_terminal) so the Dynamo adapter can run the same steps; - chat_completion_full_generator records each output, restores the reply's text and usage, and wraps the parser; - the generation-cut route, and aborts that never block the loop behind a cut (abort_request_capture). Other conflicts: - pyrefly.toml, vllm/utils.py, the wiring test's imports: both sides. - test_capture_route_tensors.py, test_vllm_token_capture_hosting.py: NVIDIA-NeMo#4506's side, which builds the worker with _init_token_capture_state. What NVIDIA-NeMo#4506 changed under this branch's code: - the Dynamo adapter and its test hosts use the capture registry that replaced _capture_calls; - two of NVIDIA-NeMo#4506's tests build the route's request class, which is a pydantic model here: they construct it instead of calling it bare. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…NVIDIA-NeMo#4905 + NVIDIA-NeMo#4506) into jthomson04/dynamo-1.7 Each frontend now takes NeMo-Gym's generation cuts on a port of its own. FrontendHost.start picks the cut port on its node beside the HTTP port and returns both URLs (FrontendUrls). ManagedDynamoRuntime.generation_cut_control_urls lists one per frontend_urls entry, in the same order, and VllmGeneration.generation_cut_control_urls passes them to Gym, which pairs each with its base_url. publish_chat_capture sends the cut's bearer token and timeout to every frontend's capture file with the rest of the capture config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
main's squash of NVIDIA-NeMo#4905 has the same tree as 9919396, which this branch already merged, so this records it as merged with no content change (-s ours). A plain merge of main would report 21 false conflicts on the lines this branch has edited since. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
NVIDIA-NeMo#4905 removed the separate dynamo generation backend, so the dynamo branches this PR patched and their test are gone; dynamo now runs on the vLLM path, which already merges hf_config_overrides. Signed-off-by: Shubhankar Tripathy <reach2shubhankar@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…NVIDIA-NeMo#4905 squash) into jthomson04/dynamo-1.7 Two conflicts, both resolved by keeping both sides: - docs/guides/dynamo-generation.md: "Where the cut goes" keeps NVIDIA-NeMo#4979's deadline (each engine cuts up to that frontend's deadline, token_capture.control_timeout_s from the cut's arrival there) and this branch's one cut port per frontend, one per base_url. - tests/unit/models/generation/test_dynamo_generation_cut.py: imports only; NVIDIA-NeMo#4979's late-engine test runs through this branch's per-host capture publishing. With several frontends each frontend that takes a cut reads the timeout from its own node's capture pickle, which publish_chat_capture writes on every host, and so takes its own deadline. An engine runs the cuts from different frontends one after another, each against its own deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Resolve uv.lock by re-locking the branch lock against the merged pyproject (adds ai-dynamo 1.5.1 from #4905; pydantic split matches main). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Meng Xin <164952785+mxinO@users.noreply.github.com>
Resolve uv.lock by re-locking the branch lock against the merged pyproject (adds ai-dynamo 1.5.1 from #4905; pydantic split matches main). Signed-off-by: Meng Xin <164952785+mxinO@users.noreply.github.com>
…split and NVIDIA-NeMo#4979's frontend chat engine
Summary
policy.generation.dynamo: trueon a SingleController vLLM generation config is now the whole Dynamo switch. NeMo RL's own vLLM workers register theirAsyncLLMwith a manageddynamo.frontend. They do this through Dynamo's engine-agnostic backend SDK (dynamo.common.backend), from inside each worker's actor loop.The engine is the native vLLM worker, not Dynamo's
dynamo.vllm, so:_serve_rollout(refactor(vllm): share the per-sample rollout pipeline and chat token helpers #4978), the same method nativegenerate_asynccalls, and the driver assembles the output withvllm/request_pipeline.py.dynamo/engine_adapter.py(DynamoEngineAdapter) owns the request kinds, the error mapping and the completions-only registration, and imports no backend.dynamo/vllm_adapter.py(DynamoVllmAdapter) is the vLLM backend:serve_rollout,start,abortandkv_event_sources. feat(dynamo): serve NeMo-Gym through the Dynamo frontend #4979 adds a request kind and Megatron (feat(megatron): serve NeMo-Gym from the engines behind the Dynamo frontend #5097) adds a backend without touching the Dynamo side.ai-dynamo-runtimecaps pydantic at 2.13): until a Dynamo release lifts a bound, override it under[tool.uv] override-dependencies.This replaces #4725 and #4822. P/D disaggregation (#4773) is out of scope.
Split at review request (three PRs, stacked):
/v1/completions. It does not need a Gym bump.What changes for users
Turning it on:
backend: vllm+dynamo: true.dynamo_cfgis optional; every field has a default (seeexamples/configs/grpo_math_1B_dynamo_single_controller.yaml).Removed:
backend: dynamo,DynamoGeneration, thedynamo.vllmlauncher, and the HTTP refit routes, token wrapper and metrics scraper;/opt/dynamo_venv;dynamo_cfgkeysengine,control_timeout_s,metrics_{include,exclude}_prefixes,worker_argsandfrontend_args.tokenizer*.Each is rejected at startup with the migration.
Packaging:
ai-dynamoandai-dynamo-runtime1.5.1 join thevllmextra (base package only, neverai-dynamo[vllm]).aisimulate, which brings in jax/flax, andkubernetes) are dropped with overrides.ai-dynamo-runtimecaps pydantic at 2.13, so vLLM venvs resolve pydantic 2.13.0 while the rest of the lock stays on 2.13.5.BUILD_DYNAMO=1now installs only etcd and nats-server.NeMo-Gym with
dynamo: trueis rejected at startup until feat(dynamo): serve NeMo-Gym through the Dynamo frontend #4979.backend: dynamoserved Gym on the legacygrpo.pypath.grpo-nanov3-30ba3b-3n8g-megatron-dynamo-swe1, moves to feat(dynamo): serve NeMo-Gym through the Dynamo frontend #4979 with its driver and config test.dynamo_cfg.request_timeout_sdefaults to no deadline, as native has none. On the SingleController,async_rl.rollout_failure.native.generation_timeout_sbounds both paths.Rejected at startup with
dynamo: trueenv.should_use_nemo_gym), until feat(dynamo): serve NeMo-Gym through the Dynamo frontend #4979;expert_parallel_size > tensor_parallel_size(data parallelism inside vLLM, which Dynamo does not serve yet: above DP rank 0, vLLM refuses each engine'sipc://KV-event endpoint);vllm_cfg.http_server_api_key_required(the frontend serves the same engines on 0.0.0.0 and checks no bearer token);frontend_args.router_mode: direct(it needs a worker ID that rollouts never carry);dynamo: trueon a non-vLLM backend;dynamo_cfgwithoutdynamo: true.Validation
These local runs on 2x RTX PRO 6000 (sm_120) were made before the split, on the combined branch. Each is the native SingleController test unchanged, with and without
++policy.generation.dynamo=true; the Gym rows live in #4979.grpo_dp_single_controller.sh(math, 2 steps)gen_kl_error0.0006 vs 0.0005;token_mult_prob_error1.014 bothgrpo_dp_single_controller_reload_refit.shtoken_mult_prob_error1.0145/1.0131 vs 1.0151/1.0109At
1e73994f85, the same math overlay locally:No module named 'megatron'in the local driver venv. The native run fails identically there, so the local env can no longer finish the run; CI's L1 covers it.Several engines and the KV router's benefit come from a separate stack. The Oct 28 bench on tyche used ai-dynamo 1.6.1, the #4955/#4979 stack, NeMo-Gym's OpenAI path, and 3 vLLM TP1 engines plus 1 trainer:
It shows several engines registering, publishing KV events and being routed between. It does not cover this PR's own rollout path or ai-dynamo 1.5.1: both CI runners have 2 GPUs, and two engines need 3.
At scale, the production Nano 3.5 RLVR recipe ran 10 steps on 86 nodes with
dynamo: true, on the #4979 stack (the recipe uses NeMo-Gym) and ai-dynamo 1.5.1. It showed no meaningful step-time degradation compared with the recipe's native vLLM runs, and rewards tracked native (W&Bo5ag5a9zfor Dynamo,6be0a52pandids5h9ltfor native).Both paths fail fast when a generation worker is killed: a local chaos run with fleet health off stopped 7 s after the kill in both modes.
Dynamo results vary slightly run to run. Requests arrive over HTTP, so batch composition differs between runs; native repeats its first step bit for bit.
CI:
L1_Functional_Tests_Dynamo.shruns these as overlays:#4979 adds the three Gym overlays: token capture, no capture, and R3 with deferred routes.
Each overlay runs the native test's own metric checks plus
check_dynamo_overlay.sh, which checks:Labels: this needs
CI:L1. The Dynamofastentries cannot pass on main's current image until it is rebuilt with/opt/dynamo/bin.Follow-ups
route_assembly.pyaccepts only MoE-layer payloads, while vLLM reports every layer. This happens identically on native and Dynamo.BUILD_DYNAMO).configure_dynamo_logging()in embedded engines.🤖 Generated with Claude Code